The present invention relates to an image combining apparatus suitably used in an electronic camera system having a panoramic image sensing mode and, more particularly, to an image combining apparatus for generating a panoramic image by connecting a plurality of images which partially overlap each other.
As conventional image combining apparatuses for generating a panoramic image (to be also referred to as a combined image hereinafter) by combining a plurality of images obtained by sensing an object image so that the images partially overlap each other, image combining apparatuses disclosed in, e.g., Japanese Patent Laid-Open Nos. 4-52635 and 5-122606 are known.
An image combining apparatus disclosed in Japanese Patent Laid-Open No. 4-52635 detects information of, e.g., the three-dimensional position, direction, or the like of a camera apparatus, i.e., information for defining the relationship between adjacent frames or scenes, upon sensing images by an electronic camera, stores such information together with image signals, and reproduces the image signals on the basis of the stored information. With this operation, the image signals are reproduced while holding the positional relationship between adjacent frames or scenes of the sensed images, and various applications, e.g., reconstruction of an image in a three-dimensional space, and the like, can be achieved.
However, in the conventional image combination represented by Japanese Patent Laid-Open No. 4-52635, upon generating a panoramic image by combining a plurality of images, when the individual images are converted into equal-magnification images on the basis of zoom magnification information, and the converted image signals are recorded on a frame memory, information of the three-dimensional positions, directions, and the like is simultaneously recorded in correspondence with each image, and the recorded image signals are merely combined to generate a panoramic image.
More specifically, in the electronic camera system of Japanese Patent Laid-Open No. 4-52635, since a plurality of panoramically sensed images are merely pasted on the frame memory in accordance with the information of their three-dimensional positions, directions, and the like, the panoramic image obtained by combining these images is nothing but the one obtained by simply laying out the individual images. Accordingly, the joint portions become discontinuous, and a panoramic image with only poor appearance is obtained.
On the other hand, an image combining apparatus disclosed in Japanese Patent Laid-Open No. 5-122606 senses a plurality of images to be connected to partially overlap each other, and obtains the luminance difference or chromaticity difference between areas (to be referred to overlapping areas hereinafter) at the end portions of the images to be connected. Upon combination, these images are connected to make the overlapping areas overlap each other, so that the luminance difference or chromaticity difference between the overlapping areas of the images to be connected assumes 0 or a minimum value.
In this manner, the image combining apparatus is convenient since an image sensing device or the object to be sensed need not be precisely moved and aligned for the purpose of connecting a plurality of images. In addition, since the connecting position is determined using the luminance difference or chromaticity difference between adjacent images, the images can be accurately combined without impairing connectability.
In the above-mentioned image combining apparatus, when a panoramic image of a wide field angle is to be generated on the basis of a plurality of images sensed so that their image fields partially overlap each other, the following processing is executed. That is, two images are connected on a plane by performing geometric transformation such as the affine transformation for the two images, so that identical points in the overlapping areas of the two images match each other.
More specifically, when the image of an object is sensed by panning the camera in the horizontal direction, two images are obtained by sensing the image of the object so that an object-side principal point O of the camera lens does not substantially move during panning, as shown in FIG. 51.
In FIG. 51, I1 and I2 represent image sensing planes before and after panning, hi represents the horizontal field of view of the image sensed before panning, and h2 represents the horizontal field of view of the image sensed after panning. If xcex8 represents the horizontal field angle of the camera lens, images each having a horizontal field angle xcex8 are obtained from the image sensing planes I1 and I2. Therefore, if xcex1 represents the overlapping angle between the two images, the horizontal field of view of the combined image obtained by these two images is (2xcex8xe2x88x92xcex1).
Assume that a plane having a rectangular frame is placed in front of the camera. This frame is placed in front of the camera at a substantially equal angular position with respect to the two image sensing planes I1 and I2 In this case, images obtained by the two image sensing planes I1 and I2 are expected to be images a and b shown in FIG. 52. As for the image sensing plane I1, since the right side of the frame is closer to the camera, the right side of the image a is sensed to be vertically larger than its left side. Conversely, as for the image sensing plane I2, since the left side of the frame is closer to the camera, the left side of the image b is sensed to be vertically larger than its right side. More specifically, the two straight lines, which are originally parallel to each other, of the frame are sensed as two straight lines which are not parallel to each other in the images a and b.
For this reason, when the two images a and b are combined by performing the affine transformation of translation processing alone (i.e., the affine transformation which does not include enlargement and rotation in the image plane) for these two images a and b, a combined image ab including two straight lines which cross each other is obtained, as shown in FIG. 53. In this combined image ab, frame lines duplicate themselves in an overlapping portion Pab resulting in an unnatural image. This is because images cannot be accurately combined by translation in the image plane since the two image sensing planes I1 and I2 are not spatially present in a single plane. Even when image enlargement or rotation within the image plane is performed, the unnaturalness cannot be eliminated.
In order to alleviate the unnaturalness in the overlapping portion Pab produced by the image combining apparatus of Japanese Patent Laid-Open No. 5-122606, Japanese Patent Laid-Open No. 5-14751 discloses a panoramic image combining apparatus, which combines images by performing geometric transformation for projecting a plurality of images sensed to partially overlap each other onto a single columnar surface.
More specifically, the panoramic image combining apparatus of Japanese Patent Laid-Open No. 5-14751 obtains a combined image free from any unnaturalness without double frame lines in the overlapping portion Pab by temporarily projecting images a and b obtained by the two image sensing planes I1 and I2 onto a common columnar surface.
However, in the conventional image combining apparatuses in Japanese Patent Laid-Open Nos. 122606, 5-14751, and the like, since the individual images are combined by setting and connecting overlapping areas, so that the luminance difference or chromaticity difference in the overlapping portion of images assumes 0 or a minimum value, i.e., since two areas whose luminance difference or chromaticity difference assumes 0 or a minimum value are simply connected, a panoramic image obtained by combining the images is merely the one obtained by laying out the individual images, as in the case of Japanese Patent Laid-Open No. 4-52635. Therefore, the joint portions become discontinuous, and a panorama image with poor appearance is obtained. For example, when the image to be connected rotates from an original image or the enlargement factor of the original image is different from that of the image to be connected, the image quality of the combined image considerably deteriorates.
On the other hand, in the image combining apparatus of Japanese Patent Laid-Open No. 5-14751, which combines images by projecting the individual images onto a columnar surface to geometrically transform them, when a plurality of images obtained by sensing an object image while panning the camera in both the horizontal and vertical directions are combined, an unnatural image in which frame lines are doubled in the overlapping portion of a combined image is obtained like in the combined image Pab in FIG. 53, which is obtained by only translating the sensed images.
When a rectangular frame is sensed at, e.g., four framing positions using the image combining apparatus of Japanese Patent Laid-Open No. 5-14751, four images c1 to C4 are obtained, as shown in FIG. 54. When these images c1 to c4 are combined by only translating them without any projection of images onto a columnar surface, an unnaturally combined image c in which frame lines are doubled in overlapping portions Pc is obtained, as shown in FIG. 55. Even when the images c1 to c4 are combined by temporarily projecting the images onto a columnar surface and then translating them, an unnaturally combined image cxe2x80x2 in which frame lines are doubled in overlapping portions Pcxe2x80x2 is obtained, as show in FIG. 56.
This means that even when two images obtained by sensing an object image by panning the camera are to be combined in the horizontal direction, if the camera has been tilted slightly during image sensing, an unnatural image in which frame lines are doubled in the overlapping portion of the combined image is obtained.
The present invention has been made to solve the above-mentioned problems, and has as its object to provide an image combining apparatus which can always easily obtain a high-quality panoramic image.
It is another object of the present invention to provide an image combining apparatus, which stores a series of images and image sensing conditions of the series of images in correspondence with each other, so that one of a plurality of combining means can be adaptively selected on the basis of the image sensing conditions corresponding to the individual images upon combining the images.
According to a preferred aspect of the present invention, corresponding points in the overlapping portions of a series of images to be combined are detected, image sensing parameters used upon executing combining processing are generated on the basis of the detected corresponding points, and the combining processing is executed using the image sensing parameters.
According to a preferred aspect of the present invention, focal point position information upon image sensing is detected as an image sensing condition. Therefore, a storage means stores the focal point position information in correspondence with the individual images. A control means discriminates based on the focal point position information corresponding to a series of images to be combined if these images are obtained by short- or long-distance image sensing. If it is determined that the images are obtained by short-distance image sensing, the control means selects a first combining means for short distances on the basis of the discrimination result; otherwise, the control means selects a second combining means. The series of images obtained by short-distance image sensing are combined by appropriate combining processing by the first combining means, and the series of images obtained by long-distance image sensing are combined by appropriate combining processing by the second combining means for long distances.
According to a preferred aspect of the present invention, a transforming means transforms pixel values of the overlapping portions of the individual images on the basis of image sensing conditions corresponding to a series of images to be combined. With this processing, an image near the connecting portions of adjacent images is transformed.
According to a preferred aspect of the present invention, a detection means detects exposure information upon image sensing as an image sensing condition. Therefore, the storage means stores focal point position information corresponding to the individual images. The transforming means corrects the density levels of the overlapping portions of the individual images on the basis of the exposure information corresponding to a series of images to be combined.
According to a preferred aspect of the present invention, an image combining means combines a plurality of spherical projection images obtained by a spherical projection transforming means on the basis of the image sensing conditions corresponding to a series of images to be combined.
According to a preferred aspect of the present invention, the spherical projection transforming means projects and transforms the individual images onto a spherical surface having the focal point position upon image sensing as a center, thereby generating a plurality of spherical projection images to be combined.
According to a preferred aspect of the present invention, the image combining means combines a plurality of spherical projection images obtained by the spherical projection transforming means, and a plane projection transforming means projects and transforms the combined image onto a plane again to generate a plane projection combined image.
According to a preferred aspect of the present invention, an adding means adds projection plane type information indicating a spherical projection image to images obtained by the spherical projection transforming means, and adds projection plane type information indicating a plane projection combined image to an image obtained by the plane projection transforming means.
According to a preferred aspect of the present invention, the image combining means combines a plurality of spherical projection images obtained by spherical projection transforming means to obtain a combined image, and selectively outputs one of the combined image and the plane projection combined image via an output means in correspondence with the field of view of the combined image.
According to a preferred aspect of the present invention, an optical axis control means controls the directions of the optical axes of a plurality of image sensing means on the basis of image sensing conditions detected by the detection means. Accordingly, an image is divisionally sensed by the plurality of image sensing means so that the sensed images partially overlap each other.
According to a preferred aspect of the present invention, the angle of the apparatus is detected, and the detected angle information is held together with images sensed in the panoramic image sensing mode. Upon combining these images, a combining scheme is adaptively selected on the basis of the angle information. With this control, image combination is done by an optimal combining scheme with respect to the angle of the apparatus.
According to a preferred aspect of the present invention, the position of the apparatus is detected, and the detected position information is held together with images sensed in the panoramic image sensing mode. Upon combining these images, a combining scheme is adaptively selected on the basis of the position information. With this control, image combination is done by an optimal combining scheme with respect to the position of the apparatus.
In order to achieve the above objects, according to another image combining apparatus of the present invention, a condition detection means detects the attitude of the apparatus produced upon movement of the apparatus as moving state information. A storage means stores the moving state information of the apparatus detected by the condition detection means and image sensing conditions as additional information in correspondence with the sensed images. A prediction means predicts the overlapping area between adjacent images to be combined on the basis of the additional information stored in the storage means. A correspondence detection means detects corresponding points in the overlapping area predicted by the prediction means. An image combining means combines a series of images on the basis of the corresponding points detected by the correspondence detection means.
According to a preferred aspect of the present invention, the condition detection means detects an angular component of the movement of an image sensing means produced upon rotation of the apparatus during image sensing.
According to a preferred aspect of the present invention, the condition detection means detects the angular component using an angular velocity sensor.
According to a preferred aspect of the present invention, the condition detection means detects a translation component of the apparatus produced upon translation of the apparatus during image sensing.
According to a preferred aspect of the present invention, the condition detection means detects the translation component using an acceleration sensor.
According to a preferred aspect of the present invention, a position detection means detects the position of the apparatus produced upon movement of the apparatus during image sensing. The storage means stores position information obtained by the position detection means as the additional information.
According to a preferred aspect of the present invention, the position detection means detects the three-dimensional position of the apparatus produced upon movement of the apparatus during image sensing.
According to a preferred aspect of the present invention, an overlapping area detection means sequentially detects the degree of overlapping between sequentially input images by calculations on the basis of the moving state information of the apparatus detected by the condition (moving information) detection means and image sensing conditions detected by an image sensing condition detection means. A comparison means compares the degree of overlapping detected by the overlapping area detection means with a predetermined threshold value. A control means makes the operation control of the overall apparatus to set the apparatus in the image sensing state at the timing at which the control means determines based on the comparison result of the comparison means that the degree of overlapping is equal to the threshold value.
According to a preferred aspect of the present invention, an image sensing situation detection means detects the image sensing situation of a plurality of images to be combined, in accordance with the additional information stored in the storage means. When it is determined based on the detection result of the image sensing situation detection means that the plurality of images to be combined are obtained by rotation, a switching control means controls a spherical projection transforming means to generate a spherical projection image by projecting and transforming the plurality of images onto a spherical surface, and controls a switching means to select and output the spherical projection image obtained by the spherical projection transforming means.
According to a preferred aspect of the present invention, an output means outputs a panoramic image generated by the image combining means to an external device.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.